I wonder whats the best way to integrate this here is. Binding this to samplers feels wrong.

Short test, trying to find the value for cfg++ that matches the image the closest.

euler_a 5cfg	euler_a_cfg_pp 1.1cfg

I still think it's a good addition to the samplers. I tend to see more coherent generations at relatively higher CFGs, however I may be biased, this requires more testing.
When I'll have some time I'll try to create a grid showing the differences in CFGs between the sampler variants to have a better idea whether this is worth adding as a sampler or not.

From some initial testing with a SDXL model (cyberrealisticXL_V90, positive prompt: "a cute cat, best quality,", negative prompt: "worst quality,", 20 steps, seed 42, discrete scheduler) the biggest image similarity between Euler and Euler CFG++ is at 4.5 and 1.1 CFG (91.62% SSIM 61.07 MSE).

There's also a high similarity between CFG 1.4 and 5.5 (90%).

Other than that, as the CFG++ paper specifies this method allows for a smoother trajectory of generation:

Positive prompt: "a cute cat, best quality, black background",
Negative prompt: "worst quality, photo,",
Seed: 42, Scheduler: discrete

5 steps Euler CFG++	5 steps Euler	Target Image (20 steps Euler)

Next I'll also try on more complex prompts.

I did some more testing, on a more complex generation there appears to be an increase in difference in the generated images.
On the same test model (cyberrealisticXL_V90), 20 steps with positive prompt: "dog jumping towards a door, outdoors, grass, house, best quality, photography" and negative prompt: "worst quality," here's what I get:

CFG	4.0	4.5	5.0	5.5	6.0
Euler

CFG	1.1	1.2	1.3	1.4
Euler CFG++

CFG	1.5	1.6	1.7	1.8
Euler CFG++

The highest similarity in this case comes from 5.5 and 1.3 and it's still just 78%:

I can't really say if it's better or not, however given the smooth generation trajectory I think that this method may introduce less artifacts in certain kind of images (I just managed to get it to happen on very few samples, but I'm not that good at writing prompts so I can't really test that aspect so well).

As I've mentioned in #1363, a better strategy could be turning the samplers into implementation objects, so that:

if (method == EULER_CFG_PP_SAMPLE_METHOD || method == EULER_A_CFG_PP_SAMPLE_METHOD) {
    LOG_WARN("Spectrum requested but not supported for CFG++ samplers");
    return;
}

would become something like this:

if (sampler->works_with_unconditioned_output()) {
    LOG_WARN("Spectrum requested but not supported for the %s sampler", sampler->pretty_name().c_str());
    return;
}

so we keep most sampler-specific quirks inside the sampler class.

OTOH, I don't think this implementation should necessarily wait for that refactor; it's not a lot of code, and it's already well tested, so it'd be perfectly fine to adapt it afterwards.

That seems like a good change for maintanability.

I've taken a look at your PR and it sure looks cleaner without the massive switch. It also happens that, while writing these samplers, I've also been thinking a lot about how to use the eta parameter to control the noise levels so it's nice to have it implemented. I don't think it would be a problem to adapt this code to your PR even if yours gets merged before this, as the overall structure is quite clear.

I'm converting this to draft until I've reworked the code to be compatible with the latest changes.

I finally had some time to work on this and rebase it on the latest master. I'm not entirely sure if this can be considered completely good as it is right now but at least it works again.

Thank you for your contribution.